For example, in movies and other image productions, in the case where various simulations of pictures (images) are carried out, or any special effect is given to pictures, a remarked portion (object) is extracted from picture, or the extracted portion is synthesized with respect other pictures. Such a partial extraction from picture and/or synthesis (of the extracted portion) is carried out by using key signal.
As a technology relating to such a key signal, an example thereof is proposed in the specification and the drawings of the U.S. Pat. No. 4,488,169 (Registered Date: Dec. 11, 1984) by the applicant of this invention.
As the key signal, there are a key signal called hard key and a key signal called soft key. Now, in the case where, e.g., the remarked portion within picture used for generating the key signal is caused to be the foreground (foreground picture), and the portion except for the above is caused to be the background (background picture), a binary key signal (mask picture) in which the range of the foreground in the picture is caused (assumed) to be 1 and the range of the background is caused (assumed) to be 0 as shown in FIG. 1A is called hard key. On the contrary, a key signal which can take not only two values of 0 and 1, but also continuous real numbers within the range from 0 to 1 as shown in FIG. 1B in consideration of alias and/or motion blur which will be described later taking place in picture is called soft key. Here, the area where the key signal takes values within the range from 0 to 1 is called gray area as the occasion may demand.
Namely, it can be said that the hard key is a signal having a sharp boundary (border) line (signal in which inclination (gradient) of the boundary line where the key signal changes from 0 (or 1) to 1 (or 0) is sharp), and the soft key is a signal having a smooth (gentle) boundary (border) line (signal in which inclination (gradient) of the boundary line where the key signal changes from 0 (or 1) to 1 (or 0) is smooth (gentle)).
It is now assumed that a (a value) shown in FIGS. 1A and 1B indicates continuous key signal values and also indicates, in this case, values of the key signal of pixel units at the same time. In practice, e.g., continuous key signals are caused to undergo filtering by the range of pixel, etc. to use, as the key signal of the pixel unit, the result by that filtering at the center of corresponding pixel. In this case, in the picture, pixel value of the pixels at the boundary portion between the foreground and the background is a value in which respective signal components of the foreground and the background are interposed. From facts as above, it can be said that a indicates contribution ratio (factor) indicating to what degree the foreground contributes with respect to respective pixels.
Meanwhile, it is rare that in the case where both the foreground and the background are stationary, the boundary between the foreground and the background is positioned at the boundary line between pixels constituting a corresponding picture. Ordinarily, as shown in FIG. 2A, the boundary between the foreground and the background exists within the pixel having finite dimensions. However, since the pixel is the minimum unit constituting the picture, even if the boundary between the foreground and the background as described above exists within the pixel, that pixel is caused to have color or brightness (pixel value) of any one of the foreground and the background as shown in FIG. 2B. For this reason, the boundary line between the foreground and the background is drawn along the boundary line between pixels as shown in FIG. 2C. When the entirety of such a picture is viewed, unnatural flicker, i.e., alias takes place at the portion of the boundary between the foreground and the background (boundary line between pixels).
Accordingly, even if a key signal smoothly indicating the boundary line between the foreground and the background can be obtained, alias would take place in the synthetic picture obtained as the result of the fact that the key signal mentioned above is used to carry out picture synthesis.
In view of the above, as a method of preventing alias taking place in the synthetic picture, there are, e.g., methods disclosed in the Japanese Patent Application Laid Open No. 232787/1985 and the Japanese Patent Application Laid Open No. 340671/1992. In the case of the method disclosed in the Japanese Patent Application Laid Open No. 232787/1985, the hard key is used to carry out picture synthesis, and to apply low-pass filter (LPF) only to the boundary portion between the foreground and the background to allow the boundary portion thereof to be blurred to allow it to be smooth to look at. In this method, since the hard key is used, but LPF is applied to the boundary portion between the foreground and the background, it can be said that picture synthesis is carried out by using key in which the key signal is caused to have gradient (inclination), i.e., substantially soft key.
Moreover, in the method disclosed in the Japanese Patent Application Laid Open No. 340671/1992, a block in which a remarked pixel on the boundary between the foreground and the background is caused to be the center is set to calculate a key signal (soft key) from the pixel pattern constituting the foreground and the background within that block to carry out, by using such a key signal, extraction of the foreground, and synthesis of the foreground and the background to thereby reduce the alias.
However, since the soft key used in the above-mentioned method is such that value of a at the inclined (gradient) portion (the portion of 0&lt;.alpha.&lt;1 in FIG. (B)), i.e., in the gray area is uniform, there was the problem that it is difficult to cope with motion blur taking place, e.g., in the case where the foreground is moving.
The motion blur will now be briefly described. The moving picture is realized as the result of the fact that picture (still picture) is continuously displayed, e.g., in frame units, etc. For example, assuming now that a circular object moving from the left direction to the right direction within picture exists, in the case where the frame period is assumed to be a short time such that it can be considered to be infinitesimal (infinitely small) with respect to the movement velocity of that object, that object is displayed as a circular object in respective frames as shown in FIG. 3A. However, in the case where the frame period is not a short time such that it can be considered to be infinitesimal with respect to the movement velocity of the circular object, picture of frame at a certain time t.sub.2 becomes a picture in which movement of the object from time t.sub.1 of frame preceding by one to the time t.sub.2 is reflected as shown in FIG. 3B. Namely, the object within the picture of the frame at the time t.sub.2 is not circular picture image, but picture image extended (elongated) in a thin manner in the direction of that motion such that the contour is blur. Such a phenomenon is called motion blur.
It is to be noted that while motion blur takes place not only in the case where any one of the foreground and the background is moving, but also in the case where both are independently moving, in the case where only the background is moving and in the case where both are independently moving, it is possible to consider those cases similarly to the case where only the foreground is moving if the background is taken as reference. In addition, in the case where motion blur takes place, clear boundary between the foreground and the background does not exist.
As a method of generating a key signal in which motion blur is taken into consideration as described above, there are methods, e.g., the Japanese Patent Application Laid Open No. 153493/1993 and the Japanese Patent Application Laid Open No. 236347/1993. In the method disclosed in the Japanese Patent Application Laid Open No. 153493/1993, current frame and frame preceding (earlier) by one are compared with each other to thereby determine sum total with respect to the entirety of frame of change quantities (magnitudes) of pixel value of pixels constituting the foreground of the current frame (or the frame preceding by one) and pixel value corresponding to pixels of the frame preceding by one (or the current frame) to determine the gray area of the soft key from the sum total value. However, in this method, since the soft key is determined in dependency upon the above-described change quantity of pixel value (difference between pixel values of the same pixel), the direction of movement of the foreground is not taken into consideration, so whether or not correct soft key corresponding to that movement can be obtained was not clear.
On the other hand, in the method disclosed in the Japanese Patent Application Laid Open No. 236347/1993, an approach is employed to first generate hard key to shave, at respective points on the boundary between the foreground and the background, the side surface of that hard key (side surfaces of column in which the portion of .alpha.=1 is caused to be the bottom surface shown in FIG. 1A) by the elliptical cone to thereby generate soft key. In this case, the long axis and the short axis of the ellipse constituting the bottom surface of the elliptical cone is determined by motion vector of the remarked portion and the magnitude of the edge intensity. However, in this method, the direction of the edge intensity is not correctly taken into consideration. For this reason, whether or not precise or correct soft key corresponding to movement of the foreground can be obtained was not clear similarly to the above.
This invention has been made in view of such circumstances and can contemplate generating a key signal which is precisely coping with motion blur, and permits the influence of alias to be reduced.